Distributed Spectral Dimensionality Reduction for Visualizing Textual Data

We use a Spectral Clustering model to formulate a distributed implementation using SPARK of Laplacian Eigenmaps that we call Distributed Spectral Dimensionality Reduction (DSDR). We evaluate DSDR to visualize conceptual clusters of terms in textual data from 2149 short documents written by online contributors to a State Department website. We compare DSDR with PCA, MultiDimensional Scaling, ISOMAP, and Locally Linear Embedding based on the Dunn Separation Index and computation times. We find for this dataset that DSDR is faster and better preserves high-dimensional cluster structure.